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train  users  of  its  online  information  retrieval  system,  the  Defense  RDT&E  Online  System 

(DROLS). 

In  order  to  maximize  benefits  of  CAI  to  DTIC,  this  paper  examined  three  areas: 

1.  ^earning  theory  as  it  relates  to  online  retrieval  systems.  That  is,  how  do  humans 

learn  these  systems. 

2.  Learning  theory  and  its  applications  to  CAI, 

3.  An  advanced  draft  of  DTICs  in-house-developed  CAI  course  entitled  ^'Introduction  to 
DROLS  Retrieval"  in  order  to  examine  CAI  instructional  design  principles. 

Recommendations  are  presented  in  all  three  areas. 
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ABSTRACT 


The  Defense  Technical  Information  Center  (DTIC)  is  developing  a 
Computer-Aided  Instructon  (CAI)  tutorial  course  entitled,  "Introduction  to 
DROLS  Retrieval"  in  order  to  train  users  of  its  online  information  retrieval 
system,  the  Defense  RDT&E  On-Line  System  (DROLS).  This  tutorial,  a  program 
of  interactive  instruction  which  consists  of  a  dialogue  between  computer  and 
learner,  is  intended  to  serve  several  functions:  1.  It  will  present  an 
introductory  overview  of  DROLS  before  new  users  attend  classroom  training  at 
DTIC  or  regional  locations.  2.  It  will  provide  more  experienced  DROLS  users 
or  users  familiar  with  information  retrieval  systems  with  a  means  to 
supplement  or  refresh  their  knowledge  of  the  DROLS  system  and  its  databases. 

In  order  to  maximize  the  benefits  of  CAI  to  DTIC,  this  paper  examined 
three  areas: 

1.  Learning  theory  as  it  relates  to  online  information  retrieval 
systems.  That  is,  how  do  humans  learn  these  systems?  Sound  instructional 
methods  should  be  based  on  a  knowledge  of  how  people  learn. 

2.  Learning  theory  and  its  applications  to  CAI.  What  is  known  about 
learning  theory  and  applications  to  CAI,  expecially  CAI  tutorials,  since  this 
is  the  type  of  CAI  DTIC  used  in  developing  its  DROLS  course. 

3.  An  advanced  draft  of  DTIC'S  in-house  developed  CAI  course  entitled 
"Introduction  to  DROLS  Retrieval."  Instructional  design  principles  utilized 
in  the  design  of  this  course  were  examined. 

Recommendations  are  presented  in  all  three  areas. 


INTRODUCTION 


One  of  the  missions  of  the  Defense  Technical  Information  Center  (DTIC)  is 
to  develop  and  Implement  a  training  program  for  teaching  users  Its  online 
Information  retrieval  system,  the  Defense  RDT&E  On-Line  System  which  Is 
commonly  referred  to  as  DROLS.* 

Training  presently  consists  of  trainer-guided,  small-group  Instruction 
sessions  comprised  of  lectures,  "hands-on"  practice,  and  explanations  of 
relevant  supplementary  printed  materials.  Training  may  occur  at  DTIC 
headquarters  in  Alexandria,  Virginia,  or  at  regional  locations  convenient  to 
users.  Classes  are  three  or  five  full-day  sessions  depending  upon  the 
terminal  classification  of  the  users:  Classes  for  dial-up  terminal  users 
are  three  days;  classes  for  dedicated  terminal  users  are  five  days.^ 

DROLS  Retrieval  training  for  fiscal  year  1984  consisted  of  240  people 
receiving  dial-up  training  and  57  receiving  dedicated  training.-* 

In  February  1983,  DTIC  Project  190.02  was  established  entitled 
"Computer-Assisted  Instruction  (CAI)  in  DROLS  Retrieval."  The  objectives  of 
this  project  were  to: 

A.  Provide  a  capability  for  presenting  and  preparing  computer-assisted 
instruction  in  DROLS  retrieval. 

B.  Develop  two  prototype  courses. 

According  to  Project  Leader  Richard  Thornett,  the  Intent  of  the  CAI 
project  was  to  serve  several  functions:  1.  It  was  to  present  an 
introductory  overview  of  DROLS  before  new  users  attended  classroom  training 
at  DTIC  or  regional  locations.  2.  It  was  to  provide  more  experience  DROLS 
users  or  users  familiar  with  information  retrieval  systems  with  a  means  to 
supplement  or  refresh  their  knowledge  of  the  DROLS  system  and  its 
databases.^ 
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The  progress  of  Che  project  thus  far  has  been: 

A.  Two  versions  of  a  PILOT  interpreter  for  presenting  the  CAI  were 
developed:  One  runs  on  the  DTIC-ADPE  Time  Sharing  Service  hosted  by  the 
UNIVAC  1100/61  computer,  the  other  runs  on  IBM  microcomputers  and  some 
compatibles.  PILOT,  a  common  CAI  language,  was  the  authoring  software  used 
to  write  DTIC's  CAI. 

B.  The  first  prototype  course,  "Introduction  to  DROLS  Retrieval"  (75% 
completed  in-house  at  DTIC)  is  currently  being  improved  and  completed  with 
the  assistance  of  a  contractor.  The  second  course,  "Retrieval  from  the  DROLS 
Technical  Report  Data  Base"  will  be  developed  if  funding  is  restored.  Under 
contract  terms  the  contractor  is  writing  instructional  objectives  for  both 
courses  and  assisting  in  completing  the  first  course. ^  This  paper  will 
examine  only  an  advanced  draft  of  the  DTIC  in-house  developed  version  of  the 
course  "Introduction  to  DROLS  Retrieval." 


PURPOSE  OF  THIS  PAPER 


Learning  an  online  information  retrieval  system,  particularly  one  as 
complex  as  DROLS,  can  be  a  difficult  task —  as  online  searching  "is  probably 
as  complex  a  task  as  is  being  assisted  by  computer  today. 

Assisting  learning  processes  through  CAI  should  also  be  approached  with 
caution.  As  Roblyer  points  out,  in  our  zeal  to  use  CAI  we  have  been  guilty 
of  an  uncritical  approach  to  computer  use.  "The  result  has  been  an  increase 
in  computer-related  activity  but  with  little  accompanying  progress  in 
determining  what  works,  what  doesn’t,  and  why."^ 

It  is  the  overall  purpose  of  this  paper  to  examine  how  humans  learn 
online  information  retrieval  systems;  how  DTIC's  CAI  tutorial  can  best  be 
utilized  in  this  process;  and  how  DTIC's  CAI  course,  "Introduction  to  DROL'S 
Retrieval"  can  best  be  designed  for  this  process.  The  intent  of  this 
research  is  to  maximize  the  benefits  of  CAI  to  DTIC,  thus  saving  time  and 
effort  which  would  otherwise  be  spent  in  "reinventing  the  wheel." 

The  purpose  of  this  paper,  then,  is  threefold: 

1.  To  state  present  learning  theories  associated  with  learning  online 
information  retrieval  systems. 

2.  To  review  learning  theory  and  its  applications  for  DTIC's  CAI. 

3.  To  critique  the  present  DTIC  in-house  developed  version  of  the  CAI 
course  "Introduction  to  DROLS  Retrieval"  by  utilizing  and  summarizing  major 
principles  of  CAI  instructional  design.  This  critique  will  recommend  areas 
for  possible  improvements  or  modifications.  Findings  from  this  critique  may 
be  helpful  in  modifying  the  present  course  or  developing  future  CAI  courses. 
Subject  matter  and  content  analysis  will  be  deemphasized  in  favor  of 
examining  the  Instructional  design  of  the  CAI  itself. 


printed  indexes  were  also  examined:  Computer  Literature  Index,  Government 
Reports  Announcements  and  Index,  and  Library  Literature.  Other  sources  or 
materials  recommended  through  interviews  were  also  examined.  The  literature 
review  was  limited  to  the  last  three  years  in  order  to  focus  on  current 
research.  This  was  not  an  exhaustive,  comprehensive  literature  review  due  to 
time  and  resource  restrictions. 

2.  Interviews  were  conducted  with  DT1C  personnel  and  outside  sources. 

3.  DTIC’s  in-house  developed  version  of  the  CAI  course  “Introduction  to 
DROLS  Retrieval"  was  reviewed  and  examined  by  using  a  dial-up  terminal  in 
DT1C  and  accessing  this  course  through  the  UNIVAC  1100/61  computer.  A  paper 


printout  of  the  PILOT  program  was  also  examined. 
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ASSUMPTIONS 


For  purposes  of  this  paper  an  online  information  retrieval  system  is 
defined  as:  A  system  capable  of  supporting  a  human-computer  interactive 
process  in  which  the  user  is  in  direct  communication  (online)  with  a  database 
and  a  computer  on  which  it  is  loaded.  Typically,  the  searcher  uses  a 
keyboard  to  communicate  with  the  computer  through  a  series  of  typed  commands. 
The  computer  acts  on  these  commands,  retrieving  and  displaying  the  data 
requested  usually  on  a  video  terminal  or  a  printer  device.  Results  may  be 
manipulated  by  the  searcher  so  that  final  results  will  be  printed  or 
displayed  in  a  specified  format.® 

It  is  assumed  that  DROLS  represents  a  typical  online  information 
retrieval  system  and  that  research  examining  such  systems  can  be  studied  for 
useful  applications  to  the  DROLS  system. 

A  simplified  method  for  accessing,  searching,  downloading  and  post¬ 
processing  information  from  DROLS  is  currently  being  researched  through  a 

federal  interagency  project  called  the  Defense  Gateway  Information  System 
(Gateway) .  This  Gateway  refers  to  a  concept  which  would  allow  a  searcher  to 
use  one  simple  command  system  to  access,  search,  download,  and  post-process 
information  from  a  variety  of  databases  of  interest  to  the  Department  of 
Defense  Research,  Development,  Test  and  Evaulation  community.  Features  of 
the  Gateway  are  presently  being  evaluated  by  selected  Department  of  Defense 
users. 

For  purposes  of  this  paper  it  is  assumed  that  full  implementation  of 
Gateway  features  will  be  a  future  development.  That  is,  for  many  years  to 
come  there  will  be  a  continued  need  to  train  users  to  access  and  search 
DROLS.  Users  may  also  want  to  continue  to  search  DROLS  directly. 


LEARNING  THEORIES  ASSOCIATED 


WITH  ONLINE  INFORMATION  RETRIEVAL  SYSTEMS 

A  learning  theory  can  be  defined  as  “the  formulation  of  the  laws  and 
conditions  of  learning. Over  the  years,  the  process  of  learning  has  been 
investigated  primarily  by  psychologists  seeking  to  explain  how  learning 
occurs.  Learning  theories  usually  construct  the  structure  and  events  (wMch 
are  generally  thought  to  happen  in  the  central  nervous  system)  which  result 
in  the  observed  behavior  known  as  learning.  Theories  are  usually  tested 
under  controlled  observations  and  under  a  wide  variety  of  conditions.^ 

Learning  theories  are  important  because  they  serve  as  the  basis  for 
instructional  theories  which,  in  turn,  "identify  conditions  that  will 
optimize  learning. " ^  In  other  words,  “designed  instruction  must  be 
based  on  knowledge  of  how  human  beings  learn. 

Yet,  studies  which  approach  this  topic  through  research  in  human-computer 
interaction  are  very  complex  and  difficult  to  organize.^  A  number  of 
researchers  have  written  about  the  complexity  of  learning  online  information 
retrieval  systems  and  the  problems  involved. 

Borgman  produced  a  major  review  of  studies  which  examined  the  users  of 
interactive  computer  systems  focusing  on  psychological  theory  and  method. 
Psychology  was  defined  for  Borgman's  review  as  the  study  of  human 
behavior — that  is,  the  mental  and  behavioral  characteristics  of  humans  as 
they  interact  with  computers.  Borgman  states  that  most  of  the  research  is 
based  on  cognitive  psychology.^  This  branch  of  psychology  seeks  to 
emphasize  how  humans  internally  store  and  access  data. 


Borgman  reports  very  little  research  which  applies  learning  theory  to 
online  information  retrieval  systems.  Much  of  the  applied  psychological 
research  has  tended  to  concentrate  on  individual  differences,  error  behavior, 
and  psychological  models.  Psychological  models  can  be  categorized  into  three 
types:  models  of  human  performance;  conceptual  models  of  a  system  presented 
to  the  user  by  a  designer,  researcher,  or  trainer;  and  mental  models  of  a 
system  formulated  by  the  user. ^ 

Bates  reports  that  there  is  only  an  elementary  understanding  of  the 
search  process.  Furthermore,  there  is  no  agreement  about  how  searches  should 
be  conducted  or  the  implications  of  psychological  processes  in  learning 
online  searching. 

Meadow  views  online  searching  as  a  complex  task  which:  "(1)  Does  not 
follow  a  well-prescribed  sequence  of  events,  (2)  is  difficult  to  determine 
the  success  of  while  it  is  going  on,  and  (3)  is  highly  language 
dependent.  He  further  states  that  there  is  no  well-developed  theory 
ot  model  of  a  search.  "Search  technique  is  largely  heuristic  based  on  ideas 
that  seem  to  work  but  are  not  formalized  into  a  theory. ”1® 

Fenichel's  analysis  of  online  searching  research  concludes  that 
experienced  searchers  were  not  fully  utilizing  the  capability  of  online 
systems  and  that  developing  adequate  strategies  was  the  problem.^ 

Without  a  learning  theory  base,  models  are  usually  intuitively  developed 
as  ad  hoc  models  of  human  cognitive  processes.  These  process  models  help 
offer  a  better  understanding  of  how  human  cognitive  processes  might 
function. 20 

There  are  a  number  of  models  of  how  learning  might  happen  in  learning 
online  Information  retrieval  systems.  As  noted,  Borgman 's  review  indicated 
that  models  research  can  be  broken  into  three  categories.  Some  of  the  models 
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research  is  reported  here  without  an  attempt  to  classify  it  into  any  of  the 
three  categories: 

Ingwersen  outlines  a  "cognitive  information  retrieval  model”  which 
explains  how  searchers  Interact  with  an  information  retrieval  system.  This 
model  draws  on  cognitive  science  (described  as  a  blend  of  psychology, 
linguistics,  and  artificial  intelligence)  as  well  as  perception,  problem 
solving,  learning,  and  cognition.  Yet,  how  this  model  translates  into 
instructional  theory  is  not  mentioned. 21 

Marcus'  model  of  the  search  process  includes  these  five  elements: 

1.  Formalized  problem  representation. 

2.  Search  strategy  formulation. 

3.  Execution  of  search  strategies. 

4.  Evaluation  and  estimation  measures. 

5.  Search  reformulation. 

Each  element  is  further  explained  and  subdivided  into  its  component 
tasks. 22  Although  Marcus'  model  is  designed  for  an  enhanced  assistance 
system  which  would  help  users  automatically  operate  and  access  online 
information  retrieval  systems,  it  seems  likely  that  this  model  could  also  be 
useful  in  teaching  online  information  retrieval  systems,  drawing  as  it  does, 
on  human  expert  intermediaries. 

Toliver  describes  online  searching  as  a  mixture  of  factual  knowledge  and 
intuitive  skills:  There  must  be  a  knowledge  of  the  subject  area. 
Intellectual  and  mechanical  knowledge  of  constructing  a  retrieval  strategy, 
and  knowledge  of  design  philosophy  and  approaches  taken  by  the  Information 
retrieval  systems.  Teaching  approaches  have  usually  Incorporated  rote 
learning,  training  courses,  manuals,  and  practice  at  a  terminal. 


Huq  states  that  the  search  process  must  progress  through  four  stages  or 
phases  that  must  be  completed  in  exact  order: 

1.  Confidence  Phase.  The  user  must  become  familiar  with  computer 
operations,  computer  jargon,  and  searching  terminology. 

2.  Insight  Phase.  The  user  begins  to  understand  how  to  interact  with 
the  computer,  and  how  the  information  is  retrieved.  There  is  also 
understanding  of  the  form  in  which  information  will  be  presented. 

3.  Discovery  Phase.  The  user's  understanding  of  the  online  process  is 
actually  applied  to  real  online  searches.  Concepts  such  as  vocabulary 
control,  search  strategy,  and  system  capabilities  can  be  applied  to  real 
systems. 

4.  Negotiation  Phase.  With  an  understanding  of  how  the  whole  process 
works  the  user  can  then  transfer  concepts  to  other  databases  and  attempt  more 
elaborate  searches. ^ 

However,  the  unpredictability  of  online  searching  can  cause  difficulty  to 
the  novice  searcher.  Virgil  suggests  that  the  basic  mechanics  of  searching 
should  be  taught  as  well  as  specific  plans  and  approaches  should  the  search 
take  a  different  direction.  Thus  teaching  should  include  a  set  of  alternate 
responses  to  possible  search  outcomes. ^5 

Borgman  states  that  conceptual-based  model  training  seems  superior  to 
step-by-step  training  but  not  for  all  types  of  tasks.  This  means  that 
training  methods  which  use  models  of  the  system  provided  to  the  user  by 
trainers,  researchers,  and  designers  (conceptual-based  models)  produce  better 
results  by  users  than  teaching  methods  which  rely  on  procedural  or 
step-by-step  instruction  without  model  Instruction.  However,  routine  or 
simple  task  training  does  not  require  model  training. ^6 


Although  models  seem  a  logical  and  necessary  first  step.  Bates  states 
that  even  after  years  of  modeling  the  search  process  we  still  do  not 
understand  how  first-rate  searchers  achieve  their  results  or  how  to  teach 
novice  searchers. ^ 

In  summary,  learning  theory  relating  to  how  humans  learn  online 
Information  retrieval  systems  has  not  been  formalized.  Without  a  formalized 
learning  theory  base,  ad  hoc  models  are  usually  developed  which  seek  to 
depict  human  cognitive  processes.  Therefore,  models  are  an  attempt  to 
explain  how  learning  might  happen. 

A  number  of  models  seeking  to  explain  how  humans  learn  online  Information 
retrieval  systems  have  been  developed  over  the  years.  Researchers  such  as 
Ingwersen,  Marcus,  Toliver,  and  Huq  have  developed  models.  However,  there 
appears  to  be  no  agreement  on  the  "best”  model  to  depict  how  humans  learn 
online  information  retrieval  systems* 


LEARNING  THEORY  AND  COMPUTER-AIDED  INSTRUCTION 


If  a  learning  theory  for  online  information  retrieval  systems  could  be 
formally  developed  then  the  next  step  could  follow:  Steps  in  the  learning 
sequence  could  be  matched  with  the  most  appropriate  CAI.  That  is,  each  type 
of  CAI  has  a  different  purpose  and,  hence,  use. 28  The  computer  is  not 
suitable  for  all  learning  outcomes,  but  it  can  be  effectively  used  for  many 
learning  outcomes. 29 

A  number  of  researchers  have  identified  the  various  types  of  computer- 
aided  Instruction  and  the  types  of  skills  and  practical  applications  best 
taught  by  these  types. ^0  The  three  most  common  types  of  CAI  and 
applications  are: 

1.  Drill  and  Practice.  A  program  in  which  rote  information  is 
practiced.  The  computer  presents  the  student  with  exercises/problems  which 
the  student  must  perform  and  solve.  Reinforcement  is  provided.  These 
programs  are  intended  to  supplement  other  instructional  methods  by 
teaching  basic  knowledge.  This  is  the  most  frequently  used  and  least 
sophisticated  type  of  CAI. 81  For  online  instruction,  drill  and  practice 
CAI  can  be  used  to  practice  knowledge  of  command  language. 82 

2.  Tutorial.  A  program  of  Interactive  instruction  of  new  material  which 
consists  of  a  dialogue  between  computer  and  learner.  Information  is 
presented,  questions  are  asked,  responses  are  given,  and  feedback  is 
provided.  Progress  through  the  lesson  can  be  controlled  by  student  response 
or  computer  controlled  branching.  The  types  of  skills 

(procedural  knowledge  perfected  through  practice)  or  knowledge  (facts  and 
relationships  among  facts)  best  taught  are:  concrete  concepts,  abstract 

-17- 

[  PREVIOUS  PAGE  f 
IS  BLANK  \ 


concepts,  rules  and  principles  and  their  application,  problem  solving,  and 
skill  modeling.  These  terms  will  be  defined  in  the  following  pages. 

3.  Simulation.  An  interactive  program  which  models  or  imitates  a  real 
situation  or  phenomenom.  The  student  uses  the  simulation  to  solve  or  analyze 
problems,  learn  procedures,  operate  or  repair  models  of  complex  equipment, 
control  phenomena,  and  learn  actions  to  take  in  different  situations. 

Simulations  differ  from  tutorials.  Tutorials  present  information  and 
guide  the  student  through  initial  use  of  the  information  or  skills.  Usually 
this  is  accomplished  through  a  question-answer  format.  A  simulation,  on  the 
other  hand,  allows  the  student  to  actually  perform  an  action  in  a  context 
similiar  to  the  real  situation.  Simulations  provide  feedback  of  the 
learner's  chosen  actions  on  the  problem. 

Simulations  are  most  effectively  utilized  after  basic  principles  and 
concepts  are  learned  so  that  meaningful  problems  can  be  presented. ^3  In 
online  instruction,  simulations  have  been  used  for  representation  of  real 
searches,  demonstrating  search  strategies,  and  allowing  user  interaction.  The 
major  disadvantage  is  that  search  strategies  must  be  heuristically  developed 
by  only  replicating  a  small  part  of  what  a  real  system  would  do.^ 

DTIC's  Tutorial 

DTIC  developed  its  in-house  version  of  the  course  "Introduction  to  DROLS 
Retrieval"  using  PILOT  authoring  language  software.  (A  CAI  authoring 
language  is  a  programming  language  specifically  designed  to  include  commands 
useful  for  CAI.)  PILOT  supports  a  tutorial  approach  to  CAI  although  the 
types  of  CAI  are  not  mutually  exclusive.  For  example  PILOT  can  present 
simulations  such  as  DROLS  responses  to  user  input — that  is,  a  simulation  of  a 
DROLS  screen.  Drill  and  practice  programs  can  also  be  included.  However, 
PILOT  is  best  suited  for  tutorials  without  the  full  software  capabilities 
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necessary  to  create  simulations.^  Because  DTIC's  CAI  supports  a 
tutorial  approach  to  learning,  this  approach  will  be  examined  more 
carefully. 

Tutorials  can  best  be  used  to  teach  these  knowledges  and  skills:  Concrete 
concepts,  abstract  concepts,  rules  and  principles  and  their  applications, 
problem  solving,  and  skill  modeling.  These  terms  are  defined  as  follows: 

1.  Concrete  concepts  are  some  particular  class  of  objects,  events,  or 
relations  which  can  be  clearly  pointed  to  when  defined.  For  example,  birds, 
chairs,  or  planes  are  concrete  concepts. 

2.  Abstract  concepts  cannot  be  clearly  pointed  to  but  are  defined 
through  acquired  meaning.  For  example,  justice,  peace,  and  love  are  abstract 
concepts. 

3.  Rules  are  specific  relationships  between  stimuli  usually  performed 
with  regularity.  For  example,  the  rule  for  approaching  a  freeway  from  an 
access  road  states  that  the  driver  on  the  access  road  shall  yield  to  freeway 
drivers. 

4.  Principles  are  the  laws  by  which  a  thing  operates  or  the  method  of  a 
thing's  operation.  The  principle  of  the  conservation  of  energy  is  an 
example. 

5.  Problem  solving  is  the  application  of  rules  or  previous  learning  to 
an  actual  or  represented  problem  situtatlon  not  previously  encountered.  For 
example,  “How  can  oceans  be  effectively  cleaned  after  oil  spills?” 

6.  Finally,  skill  modeling  is  the  process  of  establishing  a  new  skill  by 
presenting  essential  or  selected  parts  of  the  skill  to  be  learned. 

All  of  these  skills  and  knowledges  are  taught  by  applying  established  CAI 
presentation  techniques  which  vary  according  to  the  type  of  skill  or 
knowledge  taught.  There  is  a  body  of  research  which  describes  CAI 
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presentation  techniques  in  more  detail.  The  reader  is  referred  to  books  by 
Alessi  and  Trolllp,  Steinberg,  or  Gagne  and  Briggs  for  further  Information. 
All  book  titles  are  mentioned  in  this  paper's  selected  bibliography. 

The  following  is  a  greatly  simplified  outline  of  the  process  of 
Incorporating  skills  and  knowledges  into  CAI: 

1.  The  process  begins  by  analyzing  "what”  is  to  be  taught  (determining 
course  content)  and  stating  in  precise  terms  what  is  to  be  taught  and  under 
what  conditions.  This  is  usually  accomplished  by  the  method  of  writing 
"instructional  objectives." 

2.  Instructional  objectives  can  be  analyzed  in  order  to  determine  the 
types  of  knowledge  or  skills  they  represent.  Once  a  determination  is  made  of 
the  type  of  knowledge  or  skills  to  be  taught  proper  CAI  presentation 
techniques  can  be  utilized. 

In  reality,  this  is  a  complicated  procedure.  In  the  literature  examined 
for  this  paper  no  attempts  were  found  to  fully  analyze  online  information 
retrieval  systems  by  labeling  the  types  of  knowledge  or  skills  necessary  for 
the  user  learning  a  system.  An  outside  contractor  is  currently  writing 
instructional  objectives  for  the  course  "Introduction  to  DROLS  Retrieval.” 
These  instructional  objectives  could  be  analyzed  in  order  to  determine  the 
type  of  knowledge  or  skills  represented.  As  mentioned  in  the  third  section 
of  this  paper  (Instructional  Design  Principles  in  DTIC's  CAI  Tutorial)  areas 
of  the  CAI  tutorial  design  could  be  based  on  this  analysis. 

In  summary,  the  three  major  types  of  CAI  are  Drill  and  Practice, 

Tutorials  and  Simulations.  Skills  and  knowledges  to  be  learned  should  be 
matched  to  the  most  appropriate  type  of  CAI  if  effective  learning  is  to  be 
achieved. 
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INSTRUCTIONAL  DESIGN  PRINCIPLES  IN  DTIC'S  CAI  TUTORIAL 


This  portion  of  the  paper  will  examine  DTIC's  in-house  developed  version 
of  the  CAI  tutorial  entitled  "Introduction  to  DROLS  Retrieval."  This  course 
presently  consists  of  sixteen  lessons  which  represent  an  overview  of  DROLS, 
its  databases  and  command  language.  The  course  emphasis  is  on  DROLS 
commands . 

Principles  of  instructional  design  for  CAI  tutorials  as  presented  by 
Alessi  and  Trollip  will  be  examined.  Suggestions  for  improvements  or 
modifications  of  DTIC's  tutorial  will  be  listed  as  comments  under  each 
principle.  The  term  "Instructional  design  principles”  is  used  here  to  refer 
to  those  factors  which  affect  a  CAI  program's  appearance  and  its  educational 
quality. 

Other  factors  important  to  this  evaluation  should  be  noted: 

1.  Subject  matter  (i.e.  actual  course  content  about  DROLS)  was  not 
evaluated  for  accuracy  or  reliability.  The  intent  was  to  examine  the 
principles  of  instructional  design  as  they  apply  to  DTIC's  CAI  tutorial. 

2.  While  all  key  principles  of  instructional  design  as  listed  by  Alessi 
and  Trollip  were  examined,  this  evaluation  emphasizes  those  which  could 
contribute  to  improvements  or  modifications  of  the  present  CAI  tutorial  or 
prove  helpful  in  designing  future  CAI  courses.  For  a  full  listing  of 
principles  of  instructional  design  for  CAI  tutorials  the  reader  should 
consult  this  paper's  selected  bibliography  for  books  by  Steinberg  or  Alessi 
and  Trollip. 

According  to  Alessi  and  Trollip,  instructional  design  principles  can  be 
applied  to  several  main  areas  within  a  CAI  tutorial. ^6  Seven  areas, 
on  the  next  page,  will  serve  as  the  outline  for  this  evaluation: 
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However,  instructional  objectives  are  currently  being  developed  for  this 
course  by  a  contractor.  Each  lesson  in  DTIC's  course  is  comprised  of  a 
section,  then  a  unit.  Objectives  should  be  developed  at  least  at  the  lesson 
level  for  students. 

c.  Directions  for  entering,  exiting,  moving  through  the  lesson,  and 
knowing  how  to  enter  responses  should  be  clearly  stated  and  available  at  all 
times. 

Comment:  Directions  are  encountered  after  the  fifth  short  unit  of  the  CAI 
course;  they  should  be  given  earlier,  if  possible.  Help  displays  could  be 
used  to  list  a  summary  of  user  commands (GOTO,  STOP,  etc.)  and  directions. 

d.  Other  conditions  of  the  program,  its  target  population  and  designated 
implementation,  should  be  stated  in  the  CAI  introductory  lesson  or  in 
accompanying  printed  material. 

Comment :  The  target  population  should  be  clearly  identified.  DTIC'S  CAI 
course  mentions  four  groups  of  users  who  might  benefit  from  the  course: 
Beginning  users  before  a  formal  DROLS  retrieval  presentation;  searchers 
experienced  In  retrieval  seeking  to  learn  DROLS;  previous  DROLS  trainees 
needing  refresher  training;  and  managers  seeking  an  overview  of  DROLS  in  order 
to  determine  possible  benefits  to  their  organization. 

Because  of  the  amount  of  detail  about  DROLS  and  the  length  of  the  course, 
the  present  CAI  is  probably  best  suited  for  intermediaries  (people  who  search 
the  system  for  others)  or  more  advanced  end  users.  Although  managers  seeking 
overviews  are  listed  as  part  of  the  target  population,  current  research  shows 
that  materials  should  be  designed  differently  for  end  users  than  for 
Intermediaries.  Overviews  for  managers  should  stress: 

o  Broad  conceptual  ideas  about  how  information  can  be  used  to  solve 
managerial  problems  (applications  orientation). 
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o  Concepts  about  the  system  rather  than  its  commands. 

o  A  variety  of  approaches  to  gain  their  attention. ^ 

Also,  because  the  course  requires  several  hours  to  complete  and  covers  so 
much  DROLS  content,  beginning  users  might  feel  overwhelmed.  As  part  of  the 
evaluation  process  for  this  CAI  course,  feedback  from  beginning  users  should 
be  assessed.  Course  content  modification,  or  perhaps  a  simplified  version  of 
the  DROLS  overview,  could  be  developed  specifically  for  this  group  of  users 
if  feedback  warrants  this  conclusion.  Any  information  about  possible  future 
courses,  the  cost  of  using  the  CAI,  a  phone  number  for  help  In  solving 
problems,  etc.  should  also  be  mentioned. 

e.  Supplementary  materials,  if  required,  should  be  stated. 

Comment:  The  DTIC  CAI  course  makes  reference  to  other  DTIC  developed 
materials  such  as  the  DDC  Retrieval  and  Indexing  Terminology  (DRIT)  and  the 
DROLS  Diverse  Dial-Up  Reference  Guide.  All  supplementary  materials  which  are 
necessary  when  using  this  CAI  should  be  listed  and  supplied. 

2.  Presentation  of  Information 

a.  Modes  of  presentation  should  ideally  be  varied  and  include,  if 
possible,  text,  graphics,  and  sound. 

Comment :  DTIC's  CAI  does  not  include  graphics  or  sound  due  to 
hardware/software  limitations  imposed  in  order  to  accomodate  the  needs  of  a 
wider  user  community. 

b.  Layouts  of  material  within  screen  displays  should  be  consistent  and 
attractive. 

Comment :  This  is  true  throughout  most  of  the  lessons.  However,  the 
simulation  of  the  DROLS  screen  should  be  indicated  more  strongly.  At  times 
the  DROLS  simulated  screen  was  integrated  with  the  CAI  instructions  making  it 
difficult  to  determine  when  a  line  of  information  represented  the  DROLS 
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screen  or  Che  CAI  text.  Clearly  marking  a  DROLS  simulated  screen  would  help 
end  this  confusion. 

c.  Scrolling  should  be  avoided.  This  refers  to  material  which  does  not 
stay  on  the  screen  long  enough  for  the  student  to  read  it. 

Comment:  There  are  several  instances  when  text  scrolls  off  the  screen. 

d.  Appropriate  presentations  should  be  used  for  presenting  concepts, 
rules  and  principles,  and  skills.  For  instance,  research  indicates  that 
concepts  should  be  taught  by  first  presenting  the  relevant  characteristics 
that  define  the  concept  followed  by  examples  of  that  concept.  For  example,  a 
triangle  has  three  angular  sides  joined  together  which  total  180  degrees. 
Next,  simple  noninstances  of  the  concept  are  presented.  Is  a  four-sided 
figure  a  triangle?  More  difficult  examples  of  the  concept  along  with  more 
difficult  noninstances  are  then  presented.  For  example,  within  a  grouping  of 
a  square,  rectangle,  and  acute  triangle,  pick  out  the  triangle. 

Comment :  As  stated  earlier  in  this  paper,  presentation  principles 
are  beyond  the  scope  of  this  paper.  Reference  should  be  made  to  the 
appropriate  books  mentioned  earlier  when  formulating  presentations  for 
knowledges  and  skills. 

e.  Thematic  prompts  should  be  used  when  possible.  A  thematic  prompt  is 
a  cue  related  to  the  theme  or  topic  of  the  presentation  and  is  used  to  guide 
and  give  hints  to  the  student.  For  instance,  a  prompt  for  a  lesson  about  ice 
cream  may  say  "Try  again.  The  answer  is  an  ingredient." 

Comment:  The  prompts  are  varied  throughout  DTIC's  tutorial.  Many  offer 
encouragement  or  acknowledgement.  For  example,  "Good,  let's  try  another." 
Thematic  prompts  are  also  found.  For  example,  the  learner  is  asked  to 
construct  a  simple  search  in  the  Technical  Report  File.  If  an  answer  Is 
incorrect  the  prompt  says,  "Did  you  begin  and  end  with  the  @  character?". 
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In  many  instances  the  prompt  gives  the  correct  answer.  "You  forgot  the 
$$$$SON." 

f.  Jargon  should  be  avoided. 

Comment:  There  are  several  instances  in  the  tutorial  when  jargon  is  used 
or  words  are  used  without  being  defined. 

g.  Help  screens  should  be  provided.  Help  screens  are  usually  of  two 
types:  procedural  and  informational.  Procedural  help  refers  to  help  in 
operating  the  CAI  lesson;  informational  help  refers  to  assistance  with  the 
content  being  studied. 

Comment :  Help  screens  are  being  planned  for  the  present  DTIC  CAI 

course.  Two  help  screens  which  would  be  helpful  are:  A  listing  of  all  the 
user  commands  (i.e.,  COMMENT,  SHOW,  GOTO,  TIME,  WAIT,  and  STOP)  with  a  brief 
definition  of  each  and  a  listing  of  all  the  course  lessons  which  can  be 
quickly  viewed  without  the  need  to  use  commands  or  course  numbers. 


a.  When  possible,  questions  should  test  comprehension  of  material. 
Questions  in  tutorials  usually  follow  two  forms: 

1.  Alternate-response  questions  (i.e.,  True/False,  matching,  and 
multiple  choice)  in  which  the  student  chooses  the  correct  answer  from  a  list. 
For  the  CAI  author,  these  questions  are  harder  to  write  and  easier  to  judge. 
These  questions  test  recognition  of  the  correct  answer. 

2.  Constructed-response  questions  (i.e.,  completion,  short-answer, 
essay)  in  which  the  student  produces  a  response.  When  compared  to 
alternate-response  questions  these  questions  are  easier  to  write  and  harder 
to  judge.  These  questions  primarily  test  recall  of  what  the  proper  answer 
should  be. 

Testing  recall  or  recognition  alone  is  incomplete.  Comprehension,  or 


testing  the  understanding  of  the  meaning  should  be  the  primary  aim  of 


questions.  Comprehension  can  be  tested  by  asking  these  types  of  questions: 
o  Paraphrase.  Replace  important  words  with  similiar  words, 
o  New  Applications.  A  rule  or  principle  is  applied  to  a  new 
situation.  For  example,  if  a  rule  for  ordering  documents  is  discussed  a  new 
application  might  include  ordering  microfiche. 

o  Categorical  questions.  Rules  or  principles  are  applied  to 
subordinate  or  superordinate  classes.  If  a  lesson  has  presented  information 
about  photosynthesis  in  a  plant  a  superordinate  question  would  ask  about 
photosynthesis  in  the  plant  kingdom,  while  a  subordinate  question  would  ask 
about  photosynthesis  in  a  simpler  plant. 

Comment :  The  present  CAI  course  asks  many  true/false,  and  matching 
questions.  While  some  of  the  questions  test  comprehension  by  applying  new 
applications  to  previously  learned  material,  more  comprehension  type 
questions  should  be  formulated. 

b.  The  response  prompt  for  questions  should  be  below  the  question,  in 
the  left  margin. 

c.  Students  should  be  allowed  more  than  one  try  to  answer  a  question. 

Comment :  In  DTIC's  tutorial  most  questions  allow  only  one  try  before  the 

correct  answer  is  given. 

d.  A  correct  answer  should  not  be  required  in  order  to  proceed. 

Comment :  In  several  instances  the  program  would  not  proceed  unless  a 

definite  correct  answer  was  given. 

e.  Response  formats  should  give  hints  to  the  student  about  the  expected 
form  of  response. 

Comment :  In  several  instances  the  expected  format  of  the  response  was  no 


-27- 


clear*  If  there  is  a  chance  of  ambiguity,  the  question  followed  by  the 
expected  form  of  response  In  parenthesis  should  be  given.  For  example: 

Which  of  the  following  equals  11?  (a,  b,  or  c) 

a.  5+6 

b.  1/2  x  24 

c.  5+2+5 

f.  Questions  should  appear  after  or  below  lesson  information  on  a 
particular  display.  Research  indicates  that  questions  asked  after  material 
presentation  facilitate  learning  of  all  previous  material. 

Comment :  DTIC's  tutorial  usually  follows  this  principle,  presenting 
questions  after  or  below  material  which  has  been  displayed. 

4.  Judging  Responses 

Judging  response  is  the  process  of  evaluating  an  answer  or  response  in 
order  to  give  feedback  to  the  student. 

a.  In  judging  responses,  different  word  order,  synonyms,  or  spelling 
variations  from  those  expected  should  be  allowed.  Correct  and  expected 
incorrect  answers  should  be  allowed. 

Comment :  In  general,  DTIC's  tutorial  requires  a  correct  answer  as 
determined  by  the  program.  Variations  in  responses,  such  as  synonyms  or 
spelling  variations  would  produce  an  incorrect  answer  prompt. 

b.  The  student  should  be  given  as  much  time  as  desired  for  responses. 

Comment :  Five  minutes  are  allowed  before  a  warning  message  appears  if 

time  exceeds  that  limit.  However,  time  can  be  extended  as  necessary. 

c.  If  a  student  decides  not  to  give  a  response  or  is  confused,  help  or 
chance  to  escape  should  be  allowed. 


Comment :  In  the  present  tutorial  there  are  no  help  screens  and  no 


established  procedures  to  escape.  However,  as  mentioned  earlier  help  screens 
are  being  planned. 

5.  Remediation 

Remediation  refers  to  the  presentation  of  additional  information  to  the 
student  who  is  not  effectively  learning.  There  are  usually  two  remediation 
techniques:  repeating  information  already  seen  and  restating  information  in  a 
new  or  simpler  version. 

Remediation  should  be  recommended  to  the  student  if  there  is  continued 
poor  performance. 

Comment :  The  present  DTIC  course  suggests  lessons  to  be  repeated  if 
individual  lesson  responses  are  incorrect. 

6.  Sequencing  Lesson  Segments 

Sequencing  lesson  segments  refers  to  the  order  in  which  the  lessons  are 
presented.  There  are  usually  two  methods  of  sequencing.  In  linear 
sequencing,  lessons  are  presented  in  a  preset,  assigned  order,  and  the  student 
must  go  through  lessons  in  the  exact  order  indicated.  In  branching 
sequencing,  the  sequence  may  be  altered  by  the  student  or  by  student 
performance. 

a.  The  overall  sequence  of  lessons  should  be  hierarchical  or  based  on  the 
difficulty  of  the  lessons. 

b.  Branching  sequencing  should  be  favored  over  linear  sequencing. 

Comment :  DTIC's  course  provides  for  branching  sequencing  controlled  by 

the  student. 

c.  Mature  students  should  be  allowed  to  control  sequence  of  the  lesson. 
Comment :  DTIC's  course  allows  students  to  sequence  lessons  through  the 


use  of  a  GOTO  command 


d.  The  student  should  be  allowed  to  abruptly  end  the  lesson. 

Comment:  DTIC's  course  allows  the  student  to  use  the  STOP  command  to  end 
the  lesson  at  any  time. 

7.  Ending  the  Tutorial 

a.  The  student  should  know  the  procedures  for  temporarily  ending  a 
tutorial. 

Comment :  The  beginning  lesson  In  DTIC's  course  tells  the  student  to  note 
the  unit  number  being  studied  if  the  student  wishes  to  stop  now  but  proceed 
at  a  later  time. 

b.  A  final  message  should  make  the  end  obvious  to  the  student. 

Comment :  DTIC's  course  displays  a  message  which  signals  the  end  of  the 


course. 


CONCLUSIONS  AND  RECOMMENDATIONS 


Conclusions  and  recommendations  can  be  formulated  for  each  of  the  three 
areas  discussed  in  this  paper: 

Learning  Theories  Associated  with  Online  Information  Retrieval  Systems 


Conclusions:  Based  on  the  literature  review  and  interviews  with 
researchers  in  this  area,  it  is  concluded  that  there  has  been  only  sparse 
reasearch  in  learning  theory  associated  with  online  information  retrieval 
systems.  Because  of  the  absence  of  learning  theory,  models  of  how  learning 
might  occur  usually  develop  on  an  ad  hoc  basis.  Models  are  one  of  the  methods 
that  can  be  used  for  understanding  how  humans  learn. 

Recommendations :  The  practical  dilemma  remains:  If  there  is  a  lack  of 
learning  theory  which  serves  as  the  basis  for  formulating  methods  of 
instruction,  how  should  training  methods  proceed?  This  question  was  posed  to 
Borgman  during  a  recent  Interview  in  order  to  formulate  recommendations  to 
overcome  this  problem. Recommendations  are  as  follows: 

1.  Models,  such  as  those  presented  in  this  paper,  should  be  examined. 
Those  which  best  fit  user  and  system  needs  should  be  "borrowed”  for  use. 

2.  Models  can  be  developed  following  user  recommendations  and  system 
requirements.  Trainers,  along  with  more  experienced  users  of  the  system  are 
ideal  candidates  to  help  formulate  such  learning  models.  User  input  was 
stressed  as  a  very  important  factor  in  developing  such  models. 

3.  Current  research  should  be  monitored  for  practical  applications. 
Specifically,  the  author  of  this  paper  recommends  that  DTIC  follow  a  model  for 
learning  online  information  retrieval  systems  and  incorporate  CAI  within  this 


model.  Huq’s  model,  discussed  earlier  in  this  paper,  is  well-suited  for  this 


purpose.  As  noted  earlier,  it  consists  of  four  phases: 

1.  The  Confidence  Phase  during  which  the  user  must  become  familiar  with 
computer  operations,  computer  jargon,  and  searching  terminology. 

2.  The  Insight  Phase  during  which  the  user  begins  to  understand  how  to 
interact  with  the  computer  and  how  the  information  is  retrieved.  There  is 
also  an  understanding  of  the  form  in  which  Information  will  be  presented. 

3.  The  Discovery  Phase  during  which  the  user's  understanding  of  the 
online  process  is  actually  applied  to  real  online  searches.  Concepts  such  as 
vocabulary  control,  search  strategy,  and  system  capabilities  can  be  applied  to 
real  systems. 

4.  The  Negotiation  Phase  during  which  the  user  understands  how  the  whole 
systems  works  and  is  able  to  transfer  learning  to  other  databases  and  attempt 
more  elaborate  searches. 

Depending  upon  user  needs,  CAI  could  be  used  for  phase  1  and  2  while 
phases  3  and  4  could  be  approached  through  the  classroom  “hands-on"  training. 
Whichever  way  the  CAI  Is  developed,  it  is  strongly  recommended  that  users  have 
input  into  the  CAI  development  process. 

Learning  Theory  and  Computer  Aided  Instruction 

Conclusions:  There  are  three  major  types  of  computer  aided  Instruction: 
drill  and  practice,  tutorial,  and  simulation.  Each  type  of  CAI  has  a 
different  purpose  and,  hence,  a  different  use.  The  type  of  CAI  used  should 
match  the  appropriate  knowledge  and  skills  which  are  to  be  taught. 

Tutorials  best  support  instruction  of  concrete  concepts,  abstract 
concepts,  rules  and  principles  and  their  applications,  problem  solving,  and 
skill  modeling.  Simulations  are  best  used  after  basic  principles  and  concepts 
are  learned. 


The  presentation  of  knowledge  and  skills  by  CAI  tutorials  will  vary 


according  to  the  type  of  skill  or  knowledge  to  be  represented.  Course  content 
can  be  analyzed  through  a  method  which  requires  writing  instructional 


objectives. 

Recommendations:  It  is  recommended  that  DTIC's  tutorial  approach  to 
online  instruction  support  those  types  of  knowledge  and  skills  identified  by 
the  research.  That  is,  a  tutorial  can  best  support  instruction  of  concrete 
concepts,  abstract  concepts,  rules  and  principles  and  their  application, 
problem  solving,  and  skill  modeling.  In  order  to  accomplish  this  task,  the 
instructional  objectives,  currently  being  written  by  an  outside  contractor, 
should  be  analyzed  in  order  to  determine  what  type  of  knowledge  or  skill  is 
represented  in  the  instructional  objectives.  CAI  presentations  should  then  be 
based  on  the  types  of  knowledge  or  skills  identified. 

Instructional  Design  Principles  in  DTIC'S  CAI  Tutorial 

Conclusions:  Several  researchers  have  defined  principles  of  instructional 
design  which  should  be  applied  to  CAI  tutorials.  This  paper  evaluated  an 
advanced  draft  of  DTIC's  CAI  In-house  developed  version  of  the  course 
"Introduction  to  DROLS  Retrieval"  by  examining  seven  broad  areas.  It  was 
concluded  that  some  of  the  instructional  design  principles  could  be  Improved 
or  modified  to  produce  a  better  CAI  tutorial.  These  conclusions  were  noted  as 
"Comments”  under  each  of  the  principles  that  was  discussed. 

Recommendations :  It  is  recommended  that  DTIC  examine  the  modifications 

and  improvements  suggested  through  this  evaluation  and  incorporate  those  which 
would  be  feasible  at  this  stage  of  project  development. 

The  more  important  instructional  design  principles  which  should  be 
incorporated  in  DTIC's  tutorial  Include: 

1.  Objectives  or  lesson  goals  should  be  stated. 


2.  The  target  population  should  be  clearly  defined  and  the  tutorial 
should  be  designed  for  the  defined  target  population. 

3.  Help  screens  should  be  provided. 

4.  If  a  student  decides  not  to  give  a  response  or  is  confused,  help 
should  be  available  or  a  chance  to  escape  allowed. 

5.  Supplementary  material,  if  required  in  the  CAI,  should  be  stated. 

6.  When  possible,  questions  should  be  constructed  to  test  comprehension 


of  the  material 
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